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(54) System and method for performing client-centric load balancing of multiple 
globally-dispersed servers 



(57) Presented is a system and a method for load 
balancing multiple globally-dispersed servers based on 
client-centric performance criteria. The infrastructure of 
the system includes load balancing domain name serv- 
ers (DNS-LBs) deployed in close physical proximity to 
the Internet service providers' points of presence. The 
DNS-LBs are then able to monitor the performance of 
the servers from a location close to the clients, which 
allows the DNS-LBs to select a server that will yield the 
best performance from that location for the client. A sec- 



ond level of the infrastructure utilizes domain name 
servers (DNS-Bs) that are deployed on the Internet 
backbones and regional provides. The authoritative do- 
main name servers (DNS-As) for the servers to be load 
balanced refer all name queries to these DNS-Bs. The 
DNS-Bs then refer the queries to one of the DNS-LBs 
based on a mapping of the DNS-ISP address to its phys- 
ically proximate DNS-LB. The DNS-LBthen returns the 
IP address of the server that will provide the best per- 
formance from that location. 
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Description 
TECHNICAL FIELD 

[0001 ] This invention relates generally to systems and 
methods for performing server load balancing across 
multiple globally dispersed web servers, and more par- 
ticularly relates to systems and methods for performing 
such global load balancing based on client-centric pa- 
rameters such as physical proximity, server availability, 
network latency, etc. 

BACKGROUND OF THE INVENTION 

[0002] While the Internet began in the late 1960's as 
a experimental wide-area computer network connecting 
only important research organizations in the U.S., the 
advent of the TCP/IP (Transmission Control Protocol/ 
Internet Protocol) protocol suite in the early 1980's 
fueled the rapid expansion of this network from a handful 
of hosts to a network of tens of thousands of hosts. This 
expansion has continued at an accelerating pace, and 
resulted in the mid-1 990's in the transition of the Internet 
to the use of multiple commercial backbones connecting 
millions of hosts around the world. These new commer- 
cial backbones carry a volume of over 600 megabits per 
second (over ten thousand times the bandwidth of the 
original ARPanet). This rapid expansion now enables 
tens of millions of people to connect to the Internet for 
communication, collaboration, the conduction of busi- 
ness and consumer sales, etc. This new economy ena- 
bled by the modern Internet serves a global community 
of users and businesses without borders and without 
time constraints common in the brick-and-mortar econ- 
omy. 

[0003] While it may have originally been possible to 
host a company's Web site on a single server machine, 
the shear volume of users on the Internet virtually pre- 
cludes such single server hosting in a manner that al- 
lows reliable and timely e-commerce to be conducted 
thereon. Specifically, the number of requests that may 
be handled per second by a single server is limited by 
the physical capabilities of that server. As the number 
increases, the server performance and response time 
to each individual request declines, possibly to a point 
where additional requests are denied service by the 
server that has reached its connection servicing limit. 
As further connections are attempted, serverfailuremay 
occur. To overcome this problem, many hosts have im- 
plemented multiple-server clusters for the hosting of the 
business' Websites to increasethe volume and perform- 
ance seen by clients while visiting these Websites. To 
ensure that no single server machine within a host clus- 
ter becomes overloaded, modern host clusters utilize 
server load balancing mechanisms to ensure distribu- 
tion of the client load between the available server ma- 
chines. 

[0004] While such a cluster architecture greatly im- 



proves a hosfs ability to serve an increasing number of 
clients, hosting a Web site at a single physical location, 
regardless of the number of server machines at that lo- 
cation, still suffers from network latencies caused by the 

s globally dispersed distribution of the clients who may 
connect to that single physical location from any point 
on the globe. Further, reliance on a single physical lo- 
cation for the hosting of an entire enterprise's Website 
subjects that enterprise to the possibility of failure of its 

10 ability to serve any clients if a failure at that site occurs. 
Such failures include long-term power outages, natural 
disasters, network outages, etc. 

[0005] To provide redundancy of operation, to mini- 
mize the risk of an entire enterprise's presence on the 

15 internet being lost, and to decrease network latencies 
caused by long-distance communication from globally 
dispersed clients, many enterprises have begun to uti- 
lize multiple, globally dispersed servers to host mirrored 
Websites at different points around the globe. These 

20 multiple web servers typically host an enterprise's Web 
site having identical content with all of the other globally 
dispersed servers, and are typically accessed via the 
same domain name. In this way, the probability of any 
single client located anywhere in the world of success- 
es fully reaching and being served by an enterprise's web 
server is greatly enhanced, regardless of failure or over- 
loading at any one server location. 
[0006] Since multiple physical servers positioned at 
globally dispersed locations are accessible via an iden- 

30 tical domain name, a mechanism is required to correctly 
resolve the domain name to an individual IP address to 
enable a client to connect and be served by a single web 
server. A simplistic method for returning only a single IP 
address to any particular client enabled by a Domain 

35 Name Server (DNS) that is authoritative for that domain 
name is known as a round robin system. In operation, 
the authoritative DNS simply returns one of the lists of 
available IP addresses upon query from the client's 
name server. Upon the next inquiry from a client name 

40 server, the authoritative DNS returns the next IP ad- 
dress in its list of available IP addresses. This mecha- 
nism continues until all of the available IP addresses 
have been provided in response to successive queries, 
at which point the authoritative DNS repeats from the 

^5 top of the list. 

[0007] While such a round robin scheme distributes 
the client traffic among the various servers, it does so 
without regard to server availability, capacity, physical 
proximity to the client, network latency, etc. As a result, 

50 it is possible for a client located in the same physical 
proximity with an enterprise's web server to be directed 
to a mirrored web server for that enterprise physically 
located thousands of miles away in another country and 
having a much smaller capacity and, therefore, a greatly 

55 increased network latency than the server at the client's 
proximate location. 

[0008] Recognizing the limitations of the DNS-based 
round robin mechanism, several companies have intro- 
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duced global load sharing products that purport to pro- 
vide a more performance-based mechanism for return- 
ing an IP address for a server that will yield better per- 
formance than the round robin approach provided by 
DNS. One such system redirects end user service re- 
quests to the closest server as determined by client-to- 
server proximity and/or client-to-server link latency 
(round-trip times) to achieve increased access perform- 
ance and reduced transmission costs. Unfortunately, 
such systems are typically employed at a single server 
site for the enterprise. As such, the monitoring of actual 
network latencies for any particular client to any partic- 
ular server site location is not possible. Instead, such 
systems typically simulate client traffic to the distributed 
servers to determine network latencies. Alternatively, 
such systems employ physical proximity between a cli- 
ent's location and a particular web server's location as 
the primary determining factor in returning that server's 
I P address to the client. Unfortunately physical proximity 
alone may not have much bearing on the best perform- 
ing web site for a particular client's location. As such, 
such systems cannot guarantee optimum performance 
from any particular client's location. There are systems 
that deploy load balancing agents at the various sites of 
the enterprise (not just one site) and figure out the la- 
tency to the client from each of these sites to determine 
the best one. This scheme, however, does not simulate 
the real-life situation of a client going to a server as ac- 
curately as can be done from a location close to the cli- 
ent. 

[0009] As an alternative to performing some type of 
load balancing across multiple enterprise servers, other 
systems provide local caching of Web site content for 
access by physically proximate clients. Such systems 
change the web page content of their client enterprises 
by changing the uniform resource locators (URLs) in it 
to point to the domain of the local cached content. In this 
system, name queries for the enterprise domain are 
handled by separate DNS servers for the cached con- 
tent system. Unfortunately, such systems remove con- 
tent control, at least for a short period of time, from the 
enterprise itself as its content is cached on the localized 
system. Indeed, such localized caching of Website con- 
tent duplicates the services provided by the globally dis- 
persed servers employed by the enterprise to ensure 
reliable performance to its clients. 
[0010] There exists, therefore, a need in the art for a 
system of global load balancing for globally dispersed 
servers that overcomes these and other known prob- 
lems existing in the art. 

SUMMARY OF THE INVENTION 

[0011] The inventive concepts of the instant invention 
involve a mechanism and infrastructure for performing 
global load balancing across a plurality of globally dis- 
persed Websites of a customer from a location close to 
the client. 



[0012] As discussed above, to increase system ro- 
bustness and to reduce network latencies resulting from 
servicing clients over large physical distances many 
companies have begun utilizing multiple Web servers 
5 located throughout the country, and indeed throughout 
different locations worldwide. In order to provide the 
best possible client experience, the connection loads 
need to be balanced across these multiple sites based 
on server load/availability, physical client proximity, net- 
10 work latency between the client and server, network 
costs, etc. While several companies have developed 
mechanisms to provide some form of global load bal- 
ancing, none of these current systems measure actual 
network latency from physical locations close to the var- 
15 ious clients. As a result, a particular client may be di- 
rected to a particular web server when, intact, a different 
web server may have smaller latencies and give better 
performance from the client's physical location. 
[0013] The system and infrastructure of the instant in- 
20 vention overcome this problem by performing global 
load balancing from physical locations in close proximity 
to the actual client. This system of Distributed Global 
Load Balancing (DGLB) includes a DNS with a load bal- 
ancer component (DNS-LB) located at or in close phys- 
25 ical proximity to every Internet service provider (ISP) 
POP. This DNS-LB is also preferably a client of the ISP, 
and therefore is configured with the addresses of the 
ISPs DNS (DNS-ISP). These DNS-LBs form the first lev- 
el of the DGLB DNS hierarchy. This first level exists in 
30 close proximity to the clients, and comprises potentially 
tens of hundreds or thousands of DNS-LBs to properly 
globally load balance all client locations. At a second 
level of the DGLB DNS hierarchy, a set of DNS servers 
(DNS-B) are deployed on the backbones or on regional 
35 providers (National/Regional backbones, Internet ex- 
change points). These will be typically few (likely to be 
in single digits or low tens). 

[0014] In operation, the DNS-LBs maintain current 
knowledge of the ISP's DNS address, and periodically 
40 notify the DNS-B machines about the addresses of the 
DNS-ISP servers. These regionally located DNS-B 
servers maintain a mapping of the DNS-ISP addresses 
to their corresponding DNS-LB addresses so that the 
DNS-Bs may direct requests to the proper, proximately 
^5 located DNS-LB. This proper DNS-LB provides the re- 
quired address information for the best Web server (or 
ordered list of addresses from best to worst) to the 
DNS-ISP. This DNS-ISP will cache the address informa- 
tion for the appropriate authoritative Website as deter- 
so mined by the DNS-LB for that particular client. This ad- 
dress is then provided to the client who will then direct 
its traffic to that site. 

[0015] In an alternate embodiment of the invention, 
the DNS-LB also performs the function of a caching en- 
55 gjne. In this embodiment, the DNS-B responds to the 
name query by giving the address of the DNS-LB cor- 
responding to the DNS-ISP that sent the request 
through the referral process described above. When the 
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address information is provided to the client, it sends its 
HTTP request to the DNS-LB who then acts as a proxy 
cache for the request. The DNS-LB is smart enough to 
retrieve the cacheable content from either the closest 
Website or another closer proxy server that has the con- 
tent required. This mechanism provides high perform- 
ance for client requests in a manner that is totally obliv- 
ious to the ISPs. 

[0016] In a further alternate embodiment of the inven- 
tion, the DNS-LBs also provide information about the 
best site (or ordered list) to DNS-Bs that can then re- 
spond to the name query by providing the address of 
the best site or the addresses of the sites ordered from 
best to worst. In this embodiment the DNS-LBs act as 
measurement services near the client (using various 
measured values to determine the best site based on 
policy) communicating their results to the DNS-Bs. 
[0017] Additional features and advantages of the in- 
vention will be made apparent from the following de- 
tailed description of illustrative embodiments, which pro- 
ceeds with reference to the accompanying figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] While the appended claims set forth the fea- 
tures of the present invention with particularity, the in- 
vention, together with its objects and advantages, may 
be best understood from the following detailed descrip- 
tion taken in conjunction with the accompanying draw- 
ings of which: 

Figure 1 is a block diagram generally illustrating an 
exemplary computer system on which the present 
invention resides; 

Figure 2 is a simplified infrastructure diagram illus- 
trating an embodiment of the distributed global load 
balancing system of the present invention; 
Figure 3 is a simplified symbolic address table map- 
ping diagram illustrating one aspect of the present 
invention; and 

Figure 4 is a simplified protocol packet illustration 
utilized in one embodiment of the present invention 
for communicating address mapping information. 

DETAILED DESCRIPTION OF THE INVENTION 

[001 9] Turning to the drawings, wherein like reference 
numerals refer to like elements, the invention is illustrat- 
ed as being implemented in a suitable computing envi- 
ronment. Although not required, the invention will be de- 
scribed in the general context of computer-executable 
instructions, such as program modules, being executed 
by a personal computer. Generally program modules 
include routines, programs, objects : components, data 
structures, etc. that perform particular tasks or imple- 
ment particular abstract data types. Moreover, those 
skilled in the art will appreciate that the invention may 
be practiced with other computer system configurations, 



including hand-held devices, multi-processor systems, 
microprocessor based or programmable consumer 
electronics, network PCs, minicomputers, mainframe 
computers, and the like. The invention may also be prac- 

5 ticed in distributed computing environments where 
tasks are performed by remote processing devices that 
are linked through a communications network. In a dis- 
tributed computing environment, program modules may 
be located in both local and remote memory storage de- 

10 vices. 

[0020] Figure 1 illustrates an example of a suitable 
computing system environment 1 00 on which the inven- 
tion may be implemented. The computing system envi- 
ronment 1 00 is only one example of a surtable comput- 
es ing environment and is not intended to suggest any lim- 
itation as to the scope of use or functionality of the in- 
vention. Neither should the computing environment 1 00 
be interpreted as having any dependency or require- 
ment relating to any one or combination of components 
20 illustrated in the exemplary operating environment 1 00. 
[0021] The invention is operational with numerous 
other general purpose or special purpose computing 
system environments or configurations. Examples of 
well known computing systems, environments, and/or 
25 configurations that may be suitable for use with the in- 
vention include, but are not limited to, personal comput- 
ers, server computers, hand-held or laptop devices, 
multiprocessor systems, microprocessor-based sys- 
tems, set top boxes, programmable consumer electron- 
ic ics, network PCs, minicomputers, mainframe comput- 
ers, distributed computing environments that include 
any of the above systems or devices, and the like. 
[0022] The invention may be described in the general 
context of computer-executable instructions, such as 
55 program modules, being executed by a computer. Gen- 
erally, program modules include routines, programs, ob- 
jects, components, data structures, etc. that perform 
particular tasks or implement particular abstract data 
types. The invention may also be practiced in distributed 
40 computing environments where tasks are performed by 
remote processing devices that are linked through a 
communications network. In a distributed computing en- 
vironment, program modules may be located in both lo- 
cal and remote computer storage media including mem- 
45 ory storage devices. 

[0023] With reference to Figure 1 , an exemplary sys- 
tem for implementing the invention includes a general 
purpose computing device in the form of a computer 
1 1 0. Components of computer 1 1 0 may include, but are 
50 not limited to, a processing unit 120, a system memory 
1 30, and a system bus 121 that couples various system 
components including the system memory to the 
processing unit 120. The system bus 121 may be any 
of several types of bus structures including a memory 
55 bus or memory controller, a peripheral bus, and a local 
bus using any of a variety of bus architectures. By way 
of example, and not limitation, such architectures in- 
clude Industry Standard Architecture (ISA) bus, Micro 
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Channel Architecture (MCA) bus, Enhanced ISA (EISA) 
bus, Video Electronics Standards Associate (VESA) lo- 
cal bus, and Peripheral Component Interconnect (PCI) 
bus also known as Mezzanine bus. 
[0024] Computer 110 typically includes a variety of 
computer readable media. Computer readable media 
can be any available media that can be accessed by 
computer 1 1 0 and includes both volatile and nonvolatile 
media, removable and non-removable media. By way 
of example, and not limitation , computer readable media 
may comprise computer storage media and communi- 
cation media. Computer storage media includes both 
volatile and nonvolatile, removable and non-removable 
media implemented in any method or technology for 
storage of information such as computer readable in- 
structions, data structures, program modules or other 
data. Computer storage media includes, but is not lim- 
ited to, RAM, ROM, EEPROM, flash memory or other 
memory technology CD-ROM, digital versatile disks 
(DVD) or other optical disk storage, magnetic cassettes, 
magnetic tape, magnetic disk storage or other magnetic 
storage devices, or any other medium which can be 
used to store the desired information and which can be 
accessed by computer 110. Communication media typ- 
ically embodies computer readable instructions, data 
structures, program modules or other data in a modu- 
lated data signal such as a carrier wave or other trans- 
port mechanism and includes any information delivery 
media. The term "modulated data signal" means a sig- 
- nal that has one or more of its characteristics set or 
changed in such a manner as to encode information in 
the signal. By way of example, and not limitation, com- 
munication media includes wired media such as a wired 
network or direct-wired connection, and wireless media 
such as acoustic, RF, infrared and other wireless media. 
Combinations of the any of the above should also be 
included within the scope of computer readable media. 
[0025] The system memory 130 includes computer 
storage media in the form of volatile and/or nonvolatile 
memory such as read only memory (ROM) 1 31 and ran- 
dom access memory (RAM) 132. A basic input/output 
system 133 (BIOS), containing the basic routines that 
help to transfer information between elements within 
computer 1 1 0, such as during start-up, is typically stored 
in ROM 131. RAM 132 typically contains data and/or 
program modules that are immediately accessible to 
and/or presently being operated on by processing unit 
120. By way of example, and not limitation, Figure 1 il- 
lustrates operating system 134, application programs 
135, other program modules 136, and program data 
137. 

[0026] The computer 110 may also include other re- 
movable/non-removable, volatile/nonvolatile computer 
storage media. By way of example only, Figure 1 illus- 
trates a hard disk drive 140 that reads from or writes to 
non-removable, nonvolatile magnetic media, a magnet- 
ic disk drive 151 that reads from or writes to a remova- 
ble, nonvolatile magnetic disk 152, and an optical disk 



drive 1 55 that reads from or writes to a removable, non- 
volatile optical disk 1 56 such as a CD ROM or other op- 
tical media. Other removable/non-removable, volatile/ 
nonvolatile computer storage media that can be used in 

5 the exemplary operating environment include, but are 
not limited to, magnetic tape cassettes, flash memory 
cards, digital versatile disks, digital video tape, solid 
state RAM, solid state ROM, and the like. The hard disk 
drive 141 is typically connected to the system bus 121 

10 through a non-removable memory interface such as in- 
terface 140, and magnetic disk drive 151 and optical 
disk drive 1 55 are typically connected to the system bus 
1 21 by a removable memory interface, such as interface 
150. 

? 5 [0027] The drives and their associated computer stor- 
age media discussed above and illustrated in Figure 1 , 
provide storage of computer readable instructions, data 
structures, program modules and other data for the 
computer 110. In Figure 1 , for example, hard disk drive 
20 141 is illustrated as storing operating system 144, ap- 
plication programs 145, other program modules 146, 
and program data 1 47. Note that these components can 
either be the same as or different from operating system 
134, application programs 135, other program modules 
25 136, and program data 137. Operating system 144, ap- 
plication programs 145, other program modules 146, 
and program data 1 47 are given different numbers here- 
to illustrate that, at a minimum, they are different copies. 
A user may enter commands and information into the 
30 computer 20 through input devices such as a keyboard 
162 and pointing device 161, commonly referred to as 
a mouse, trackball ortouch pad. Other input devices (not 
shown) may include a microphone, joystick, game pad, 
satellite dish, scanner, or the like. These and other input 
35 devices are often connected to the processing unit 1 20 
through a user input interface 1 60 that is coupled to the 
system bus, but may be connected by other interface 
and bus structures, such as a parallel port, game port 
or a universal serial bus (USB). A monitor 191 or other 
40 type of display device is also connected to the system 
bus 121 via an interface, such as a video interface 1 90. 
In addition to the monitor, computers may also include 
other peripheral output devices such as speakers 197 
and printer 196, which may be connected through an 
^5 output peripheral interface 190. 

[0028] The computer 1 1 0 may operate in a networked 
environment using logical connections to one or more 
remote computers, such as a remote computer 1 80. The 
remote computer 1 80 may be another persona! compu- 
50 ter, a server, a router, a network PC, a peer device or 
other common network node, and typically includes 
many or all of the elements described above relative to 
the personal computer 110, although only a memory 
storage device 181 has been illustrated in Figure 1 . The 
55 logical connections depicted in Figure 1 include a local 
area network (LAN) 1 71 and a wide area network (WAN) 
173, but may also include other networks. Such net- 
working environments are commonplace in offices, en- 
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terprise-wide computer networks, intranets and the In- 
ternet. 

[0029] When used in a LAN networking environment, 
the personal computer 1 1 0 is connected to the LAN 1 71 
through a network interface or adapter 1 70. When used 
in a WAN networking environment, the computer 110 
typically includes a modem 1 72 or other means for es- 
tablishing communications over the WAN 1 73, such as 
the Internet. The modem 172, which may be internal or 
external, may be connected to the system bus 121 via 
the user input interface 1 60 ; or other appropriate mech- 
anism. In a networked environment, program modules 
depicted relative to the personal computer 110, or por- 
tions thereof, may be stored in the remote memory stor- 
age device. By way of example, and not limitation, Fig- 
ure 1 illustrates remote application programs 1 85 as re- 
siding on memory device 1 81 . It will be appreciated that 
the network connections shown are exemplary and oth- 
er means of establishing a communications link be- 
tween the computers may be used. 
[0030] in the description that follows, the invention will 
be described with reference to acts and symbolic repre- 
sentations of operations that are performed by one or 
more computer, unless indicated otherwise. As such, it 
will be understood that such acts and operations, which 
are at times referred to as being computer-executed, in- 
clude the manipulation by the processing unit of the 
computer of electrical signals representing data in a 
structured form. This manipulation transforms the data 
or maintains it at locations in the memory system of the 
computer, which reconfigures or otherwise alters the op- 
eration of the computer in a manner well understood by 
those skilled in the art. The data structures where data 
is maintained are physical locations of the memory that 
have particular properties defined by the format of the 
data. However, while the invention is being described in 
the foregoing context, it is not meant to be limiting as 
those of skill in the art will appreciate that various of the 
acts and operation described hereinafter may also be 
implemented in hardware. 

[0031] The distributed global load balancing (DGLB) 
system of the instant invention is illustrated in the sim- 
plified infrastructure diagram of Figure 2 to which spe- 
cific reference is now made. The environment into which 
the DGLB of the instant invention is utilized includes a 
business enterprise having multiple server locations 
200, 202, etc. positioned at globally-dispersed locations 
to host the content of the enterprise's Web site. While 
only two separate sites 200, 202, are illustrated in Figure 
2, one skilled in the art will recognize that additional 
server sites may be included as determined by an en- 
terprise's client base and performance criteria it wishes 
to achieve. There also exists an authoritative domain 
name server (DNS-A) 204 that is capable of providing 
the IP address information for the enterprise's server 
sites 200, 202 upon inquiry from an Internet service pro- 
vider's DNS (DNS-ISP) 206 on behalf of a client 208. 
Also existing in this environment is the root domain 



name server 210, and possibly intermediate domain 
name servers (not shown) that, through successive in- 
quiries well known in the art, will eventually refer the 
DNS-ISP 206 to the DNS-A 204 for the enterprise's Web 

5 site sought by the client 208. 

[0032] The distributed global load balancing system 
of the instant invention adds to this environment an in- 
frastructure of multiple load balancing domain name 
servers (DNS-LBs) 212, 214, 216, etc. Each of these 

10 DNS-LBs 212, 214, 216, etc. are located in close phys- 
ical proximity to each Internet service provider's point of 
presence (POP) to which a client 208 can connect with 
a local telephone call. In this way, each DNS-LB is in 
close physical proximity to each client 208 being served 

15 by that particular ISP. As will now be apparent, this em- 
bodiment of the invention utilizes one DNS-LB per 
DNS-ISP. If the ISP chooses to service several POPs 
with one DNS-ISP, an embodiment of the invention will 
provide one DNS-LB for all those POPs. However, since 

20 jt is expected that the DNS-ISP would be close to the 
POPs it is serving, the DNS-LB will also be close to the 
clients served by these POPs. 

[0033] These multiple DNS-LBs 212, 214, 216, etc. 
form the first level of the DNS hierarchy of the instant 

25 invention, one that is in close physical proximity to the 
clients. As will be well appreciated by those skilled in the 
art, the number of DNS-LBs may number in the tens of 
hundreds or thousands to cover all client locations 
throughout the world. These DNS-LBs are preferably cli- 

30 ents of the ISP, and will therefore be configured with the 
address of the ISP's domain name server (DNS-ISP) 
206. In this way, the DNS-LBs will be informed of any 
address change of the ISP's DNS 206 from the ISP. 
[0034] The second level of the DNS hierarchy provid- 
es ed by the distributed global load balancing system of the 
instant invention comprises a set of DNS servers 
(DNS-Bs) 218 deployed on the Internet backbones 
(Sprint, MCI, AT&T, UUNET, etc.) or on regional provid- 
ers by agreement with these carriers (Regional back- 

40 bones, Internet exchange points). These DNS-Bs 218 
receive address mapping information from each of the 
DNS-LBs 212, 214, 216, etc. to associate these load 
balancing domain name servers with their physically 
proximate DNS-ISPs 206. These DNS-Bs 218 also re- 

45 ceive information from the authoritative domain name 
servers (DNS-As) 204 for the various enterprises who 
have chosen to utilize the services provided by the dis- 
tributed global load balancing system of the instant in- 
vention. This information includes the IP addresses of 

50 the various globally distributed server sites 200, 202, 
etc. that host the enterprise's Web site content. These 
DNS-Bs 21 8 provide their IP address to the DNS-As 204 
so that proper referral may be made to the distributed 
global load balancing system upon inquiry for the IP ad- 

55 dress of the one of the enterprise's Web sites. 

[0035] Having now described the basic infrastructure 
of the distributed global load balancing system of the 
instant invention, the operation of the DGLB will be de- 
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scribed with continuing reference to Figure 2. As dis- 
cussed briefly above, once an enterprise decides to uti- 
lize the distributed global load balancing system of the 
instant invention, that enterprise's Web site IP name to 
address mapping information is communicated from the 
authoritative DNS (DNS-A) 204 to the backbone de- 
ployed domain name servers (DNS-B) 218, etc. These 
DNS-Bs 218, etc. also provide the authoritative DNS-A 
204 IP address information that the DNS-A may provide 
in response to a query for IP address information for its 
Web sites. This communication of information may take 
place interactively as illustrated by the communication 
line 220, or may take place off-line as desired. An ad- 
vantage of providing on-line communication between 
these DNS servers 218, 204 is that changes in IP ad- 
dress information for an existing customer or for a new 
customer wanting to have its sites globally load bal- 
anced may be communicated without the delays nor- 
mally associated with off-line updates. Once the back- 
bone deployed domain name servers 21 8, etc. have the 
IP address information for the various contracting enter- 
prises' Web sites, this information may be communicat- 
ed to the numerous first-level load balancing domain 
name servers (DNS-LB) 212, 214, 216, etc, via commu- 
nication lines 222, 224, and 226. This information may 
be multicast to all of the load balancing domain name 
servers in the first level of the DGLB infrastructure, or It 
may be unicast to only particular load balancing domain 
name servers to whom an inquiry is being referred. One 
skilled in the art will recognize that information provided 
to DNS-Bs by DNS-As and to DNS-LBs by DNS-Bs can 
be cached by the respective DNSs until they get an up- 
date due to subsequent changes in sites addresses. 
The load balancer on DNSLBs will proactively check the 
health/availability and network latency of the sites peri- 
odically, e.g. every few minutes (or other period that is 
configurable), or upon receiving a query. Communica- 
tion between DNS-As, DNS-Bs, and DNS-LBs asshown 
by communication lines 220, 222, 224, 226 can be- 
through a reliable communications protocol such as 
TCP, or through some other communications protocol 
as desired. 

[0036] Once the requisite domain name servers in the 
first and second level of the DNS hierarchy of the DGLB 
acquire the IP address information of the contracting en- 
terprise's Web site server locations, the load balancing 
domain name servers 212, 214, 216, etc. must commu- 
nicate to the backbone deployed domain name servers 
218, etc. mapping information relating their IP address 
to their physically proximate Internet service provider's 
domain name server's IP address. By providing such 
mapping information to the backbone deployed domain 
name servers 218, etc., these DNS-Bs 21 8, etc. are ca- 
pable of properly referring IP address inquiries to the 
load balancing DNS that is most closely located to the 
DNS-ISP and therefore the client from whom the IP ad- 
dress request has originated. 

[0037] This mapping information may be provided 



from the load balancing domain name servers 212, 214, 
21 6, etc. via the illustrated communication connections 
222, 224, 226, etc. As will be recognized from the fore- 
going description, this mapping information needs to be 

5 communicated from each of the physically proximate 
load balancing domain name servers in the first level of 
the DNS hierarchy to each of the backbone deployed 
domain name servers in the second level. Each of these 
backbone deployed domain name servers 218, etc. will 

io utilize this information to construct and maintain a map- 
ping table such as that illustrated in simplified form in 
Figure 3. As may be seen from this simplified mapping 
table of Figure 3, the load balancing domain name serv- 
er's IP address 228 is related to the ISP's domain name 

*5 server's IP address 230. While Figure 3 illustrates sym- 
bolic representations of the IP addresses of the various 
load balancing and ISP domain name servers, one 
skilled in the art will recognize that the actual IP address 
is utilized to provide the proper mapping for referral of 

20 client-originated requests to the proper load balancing 
domain name server in closest physical proximity there- 
to. 

[0038] This information may be provided from the load 
balancing domain name servers 212, 214, 216, etc. to 

25 the various backbone deployed domain name servers 
218, etc. by transmitting a simple packet of information 
such as that illustrated in Figure 4. If this information is 
to be transmitted utilizing TCP/IP, the packet may in- 
clude the IP header 232 that contains the source ad- 

30 dress of the load balancing domain name server and the 
destination address of the backbone deployed domain 
name server. In the TCP header section 234 of this ex- 
emplary packet, the source port of the load balancing 
DNS and the destination port of the backbone deployed 

35 DNS may be included. Finally, this exemplary packet in- 
cludes a map protocol header 236 that includes the IP 
address of the load balancing DNS and the IP address 
of the ISP's domain name server associated with that 
particular load balancing domain name server. 

40 [0039] Returning again to the infrastructure diagram 
of Figure 2, the method of providing global load balanc- 
ing across multiple, globally dispersed server locations 
that host an enterprise's Web site information will now 
be described. Upon initial inquiry 238 from a client 208 

45 for the IP address of a particular Web site address, e.g. 
www.foobar.com, that client's ISP domain name server 
206 checks to determine whether it can resolve the IP 
address itself. If the DNS-ISP 206 cannot resolve the IP 
address, it queries 240 the root server 210 for the IP 

so address. The root DNS 21 0 will then refer the DNS-ISP 
206 (possibly through one or more referrals) to the foo- 
bar enterprise's DNS that is authoritative for foobar.com 
(DNS-A) 204. The DNS-ISP 206 will then query 242 the 
DNS-A 204 for the IP address-for foobar.com. Instead 

55 of returning the IP address, DNS-A 204 will again refer 
the DNS-ISP 206 to the DNS-B 218 through a delega- 
tion record. Once this referral is received : the DNS-ISP 
206 will query 244 the DNS-B 218 for the IP address for 
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foobar.com. Again, instead of returning the IP address 
for foobar.com to the DNS-ISP 206, the DNS-B 21 8 will 
referthe DNS-ISP 206 to a load balancing domain name 
server in accordance with the mapping table stored 
therein (see Figure 3). In the illustrated example, this 
referral will provide the IP address for the DNS-LB 21 2. 
The DNS-ISP 206 will then query 246 the DNS-LB 212 
for the IP address for foobar.com. This is done through 
the DNS CNAM E mechanism. That is, DNS-B 21 8 maps 
www.foobar.com to <anylabel>. www.foobar.com 
through CNAME RR type. It, therefore, redirects (refers) 
the DNS-ISP 206 to the closest DNS-LB 21 2 for <any- 
label> .www.foobar.com. 

[0040] The DNS-LB 21 2 knows which foobar.com site 
of the several that exist is most well equipped at that 
particular time to handle the request from that client 208 
location. This information is acquired by periodically 
checking the response time of the sites by performing 
HTTP operations against it. The load balancing domain 
name servers employ various characteristics and crite- 
ria to determine this information, including response 
time, to determine which of the several available sites 
should service the client's request from that physical lo- 
cation. The DNS-LB 212 then returns the IP address for 
the selected site to the DNS-ISP 206. The DNS-ISP 206 
caches that request or a time-to-live (TTL) that is re- 
turned with the query response from the DNS-LB 212. 
The DNS-ISP 206 then returns 248 this address to the 
client 208. The client 208 is then able to direct its traffic 
to the particular server site that has been determined to 
provide it with the best operating characteristics by the 
DNS-LB 212 located in close physical proximity to it. 
[0041] In this way, the client 208 is directed to a par- 
ticular server site that will provide it the lowest network 
latency (enhanced performance), that results in the low- 
est cost for the content delivery, that is in the closest 
physical proximity, or that is a combination of any or all 
of the above as determined by the enterprise policy. 
These performance measurements may utilize well 
known mechanisms including the downloading of web 
pages, determining the number of resets and abnormal 
terminations, and other various known mechanisms 
available in the art. However, unlike current systems that 
utilize these mechanisms : the infrastructure provided by 
the DGLB of the instant invention allows these perform- 
ance measures to be conducted at physical locations in 
close proximity to the individual clients, thereby provid- 
ing the most accurate measure of performance as will 
be seen by that particular client from his physical loca- 
tion. 

[0042] Since, as described above, the referral proc- 
ess happens every TTL, it does not unduly burden the 
IP address resolution to add two more domain name 
servers (DNS-B and DNS-LB) to the referral chain. The 
referral to the backbone deployed domain name server 
has a long TTL, such as, for example, one day, while 
the referral to the DNS-LB has a shorter TTL, such as, 
for example, one hour. The actual IP address returned 



by the DNS-LB has a very short TTL, such as 5 minutes, 
so that subsequent client requests will be referred to a 
particular server site that is currently providing the opti- 
mum performance. Through this mechanism, the ISP is 

5 totally oblivious to the presence of the DNS-LBs. The 
system of the invention refers queries for load balanced 
sites to DNS-LB through the normal DNS referral mech- 
anism to resolve an IP address, which allows the 
DNS-LB to gain control of how the request is answered. 

10 [0043] While the embodiment of the infrastructure of 
the DGLB of the instant invention shown in Figure 2 il- 
lustrates separate DNS-ISP and DNS-LB components, 
one skilled in the art will recognize that the functionality 
provided by these two components may be combined 

15 into a single DNS-ISP (DNS-ISP-LB). As such, the 
DNS-ISP-LB would become authoritative for the Web 
sites who have contracted for the global load balancing 
from the client location through the system of the instant 
invention. DNS-ISP-LB would receive the information 

20 concerning the various IP addresses for the particular 
Web sites from the DNS-Bs initially, or as an inquiry is 
received from a client for that information as discussed 
above for the non-combined case. 
[0044] For the DNS-ISP-LB case (where DNS-ISP 

25 and DNS-LB are combined) it is possible for the 
DNS-ISP-LB to serve multiple POPs that are not close 
to theDNS-ISP-LB's location. To allow the DNS-ISP-LB 
to perform metrics from a location closer to the POPs 
than its own location, the DNS-ISP-LB can utilize Meas- 

30 urement Service Agents (MService) located close to the 
POPs (there can be one MService per POP or for a set 
of POPs that are close to it). The performance metrics 
can be communicated to the DNS-ISP-LB (or retrieved 
by the DNS-ISP-LB) by each MService periodically, e. 

35 g. every 5 minutes (or other configurable period), or 
when the DNS-ISP-LB receives a query. 
[0045] In the embodiment where the metrics are com- 
municated/retrieved periodically, the DNS-ISP-LB will 
use the most recently received performance metrics 

40 from the MService that is close to the client's POP to 
determine which site's address to return to the client's 
address query. The DNS-ISP-LB determines the closest 
MService to the client's POP by matching the addresses 
of the MServices against that of the client. Since each 

45 MService will be a client of the POP, its address will be 
from the same address prefix as the other clients of the 
same POP, allowing for a match. One skilled in the art 
will recognize that other matching mechanisms may be 
used as appropriate. For example, the DNS-ISP-LB 

50 could maintain a map of client IP prefixes from the var- 
ious POPs and the addresses of the MService agent for 
those prefixes or POPs. This mapping table would be 
similar to the table maintained by DNS-B discussed 
herein. 

55 [0046] As a further alternative embodiment, for the 
non-combined case, the DNS-LBs could send the Web 
site response information to the DNS-Bs so that they 
may directly respond to an inquiry from a particular client 
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with the proper Web site location that will provide that 
client the best performance from his physical location. 
The information provided from the DNS-LBs could be a 
listing from best to worst of the server site IP addresses, 
or only the current best IP address as desired. In this 
embodiment, the DNS-LBs really are not performing a 
DNS service, but instead are monitoring the perform- 
ance of the contracted server sites from locations in 
close proximity to the clients at that physical locale. It is 
noted that while best performance will be achieved by 
providing a DNS-LB at each ISP POP, acceptable per- 
formance may well be achieved by deploying fewer 
DNS-LBs providing more regional than local perform- 
ance measure. 

[0047] As a further alternative embodiment, the 
DNS-LBs could also perform the function of a caching 
engine. In this embodiment, the DNS-Bs respond to the 
name query by returning the IP address of the DNS-LB 
corresponding to the DNS-ISP that sent it the request 
(through the referral process) as the address for www. 
foobar.com. Alternatively, the DIMS-B refers the 
DNS-ISP t the DNS-LB and the DNS-LB returns its own 
address instead of the address of the best performing 
site. The client 208 then sends its HTTP request to the 
DNS-LB. When that DNS-LB gets the HTTP request, it 
acts as a proxy cache for the request. Since the DNS-LB 
includes the ability to measure the performance from 
that physical location to the various server sites, it re- 
trieves the cacheable content from either the closest or 
best performing foobar site or another closer proxy serv- 
er that has the content, which is providing the best net- 
work latency. In the combined case (DNS-ISP-LB), the 
DNS-ISP-LB would return the address of the closest 
M Service that is acting as a cache. 
[0048] In view of the many possible embodiments to 
which the principles of this invention may be applied, it 
should be recognized that the embodiment described 
herein with respect to the drawing figures is meant to be 
illustrative only and should not be taken as limiting the 
scope of invention. For example, those of skill in the art 
will recognize that the elements of the illustrated embod- 
iment shown in software may be implemented in hard- 
ware and vice versa or that the illustrated embodiment 
can be modified in arrangement and detail without de- 
parting from the spirit of the invention. Therefore, the 
invention as described herein contemplates all such em- 
bodiments as may come within the scope of the follow- 
ing claims and equivalents thereof. 



Claims 

1 . A system for performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
an ISP having a domain name server (DNS-ISP), 
the servers further having an authoritative domain 
name server (DNS-A) associated therewith, the 



system comprising: 

a first domain name server deployed on an In- 
ternet backbone (DNS-B); and 
a plurality of load balancing domain name serv- 
ers (DNS-LBs) deployed in close physical prox- 
imity to the clients, the DNS-LBs having stored 
therein IP address information of the multiple 
globally-dispersed servers to be load balanced, 
the DNS-LBs each sending mapping informa- 
tion to the DNS-B relating the DNS-LB's IP ad- 
dress to an IP address of the DNS-ISP to which 
it is in close physical proximity, the DNS-LBs 
determining performance characteristics of 
each of the multiple globally-dispersed servers. 

2. The system of claim 1 , wherein the DNS-B stores 
the mapping information for the plurality of 
DNS-LBs to forward IP address queries to one of 

20 the DNS-LBs closest to the DNS-ISP from which the 
IP address query originated, and wherein the 
DNS-LB closest to the DNS-ISP returns the IP ad- 
dress to the DNS-ISP of the server having the best 
performance characteristics. 

25 

3. The system of claim 1 , wherein the DNS-B stores 
the mapping information for the plurality of 
DNS-LBs to forward IP address queries to one of 
the DNS-LBs closest to the DNS-ISP from which the 

30 ip address query originated, and wherein the 
DNS-LB closest to the DNS-ISP returns the IP ad- 
dress of the DNS-LB to the DNS-ISP. 

4. The system of claim 1 , wherein the DNS-B provides 
35 its IP address information to the DNS-A to enable 

the DNS-A to forward IP address queries to the 
DNS-B. 

5. The system of claim 4, wherein the DNS-B receives 
to |p address information from the DNS-A for the serv- 
ers to be load balanced. 

6. The system of claim 1 , wherein the DNS-LB is a cli- 
ent of the DNS-ISP. 

45 

7. The system of claim 1 , further comprising a DNS-B 
deployed on each Internet backbone, and wherein 
each DNS-B contains the mapping information for 
all of the DNS-LBs stored therein. 

50 

8. The system of claim 1 , wherein the DNS-LB trans- 
mits updated mapping information upon a change 
of an IP address of the DNS-ISP. 

55 g. The system of claim 1, wherein each of the 
DNS-LBs transmit performance information of the 
servers to the DNS-B, and wherein the DNS-B uti- 
lizes the mapping information to determine the 
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proper DNS-LB performance information to utilize 
to select the I P address of the server having the best 
performance characteristics .to return to the 
DNS-ISP from which an IP address query originat- 
ed. 

10. A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
an ISP having a domain name server (DNS-ISP), 
the servers further having an authoritative domain 
name server (DNS-A) associated therewith, the 
method . comprising the steps of: 

receiving IP address information from the 
DNS-A for the servers to be load balanced; 
providing the IP address information to a plu- 
rality of load balancing domain name servers 
(DNS-LB); 

receiving mapping information associating 
DNS-ISP IP address information to IP address 
information of a proximately located DNS-LB 
capable of determining server performance 
from a location physically proximate to the 
ISP's point of presence; and 
referring address inquiries from a DNS-ISP to 
a physically proximate DNS-LB in accordance 
with the mapping information. 

1 1 . A computer-readable medium having computer ex- 
ecutable-instructions for performing the steps of 
claim 10. 

12. A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
an ISP having a domain name server (DNS-ISP), 
the servers further having an authoritative domain 
name server (DNS-A) associated therewith, the 
method comprising the steps of: 

obtaining, by a load balancing domain name 
server (DNS-LB), IP address information for a 
DNS-ISP located in close physical proximity to 
the DNS-LB; 

providing a mapping of an IP address of the 
DNS-LB to the IP address information of the 
DNS-ISP to an external domain name server; 
receiving IP address information for the serv- 
ers; 

monitoring performance of the servers at the re- 
ceived IP addresses; and 
providing at least one IP address for a server 
in response to a name query selected based on 
the monitoring step. 

13. The method of claim 12, further comprising the 
steps of: 



detecting a change in the DNS-ISP IP address; 
and 

updating the mapping of the IP address of the 
DNS-LB to the IP address information of the 
5 DNS-ISP to the external domain name server. 

14. The method of claim 12, further comprising the 
steps of 

receiving selection criteria for the selection of 
10 an IP address; 

receiving a name query from the DNS-ISP; 

and 

wherein the step of providing at least one IP 
address for a server in response to a name query 
15 selected based on the monitoring step further com- 
prises the step of providing at least one IP address 
for a server in response to a name query selected 
based on the monitoring step and on the selection 
criteria. 

20 

1 5. A computer-readable medium having computer-ex- 
ecutable instructions for performing the steps of 
claim 12. 

25 16. A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
an ISP having a domain name server (DNS-ISP), 
the servers further having an authoritative domain 

30 name server (DNS-A) associated therewith, the 
method comprising the steps of: 

receiving an IP address of a domain name serv- 
er to which name queries are to be forwarded 
35 forjoad balancing; 

providing I P address information forthe servers 
to the domain name server; and 
forwarding name queries from the DNS-ISP to 
the IP address of the domain name server. 

40 

17. A computer-readable medium having computer-ex- 
ecutable instructions for performing the steps of 
claim 1 6. 

45 18. A computer-readable medium having stored there- 
on a data structure, comprising: 

a first data field containing IP address informa- 
tion for an Internet service provider's domain 
so name server (DNS-ISP); 

a second data field containing IP address infor- 
mation for a load balancing domain name serv- 
er (DNS-LB); and 

wherein the first data field and the second data 
55 field are associated to one another so as to pro- 

vide a mapping from the IP address of the 
DNS-ISP to the IP address of the DNS-LB. 
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. The computer-readable medium of claim 1 7, where- 
in the first data field comprises a first column of data 
fields containing a plurality of IP address entries for 
a plurality of DNS-ISPs, wherein the second data 
field comprises a second column of data fields con- 5 
taining a plurality of IP address entries for a plurality 
of DNS-LBs, and wherein an association between 
each entry in the first column and each entry in the 
second column forms mapping rows from each 
DNS-ISP IP address to a DNS-LB IP address. w 

. A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
Internet service providers (ISPs) at a point of pres- 15 
ence (POP), each ISP having a domain name serv- 
er (DNS-ISP), the servers further having an author- 
itative domain name server (DNS -A) associated 
therewith containing information regarding the IP 
addresses of the servers, the method comprising 20 
the steps of: 

deploying a first plurality of load balancing do- 
main name servers (DNS-LBs) in close physi- 
cal proximity to the ISP POPs; 25 
deploying a second plurality of second level do- 
main name servers (DNS-Bs) on the Internet 
backbones and regional provides; 
communicating IP address information for the 
DNS-Bs to the DNS-As to enable the DNS-As 30 
to refer name queries to the DNS-Bs; 
providing, by the DNS-LBs to the DNS-B, map- 
ping information associating an IP address of 
the DNS-LB to an IP address of the physically 
proximate DNS-ISP to enable the DNS-B to re- 35 
fer name queries from a DNS-ISP to the phys- 
ically proximate DNS-LB; and 
communicating IP address information of the 
servers to the DNS-LBs; 

monitoring, by the DNS-LBs at a location phys- 40 
ically proximate to the ISP POP, performance 
of the servers; and 

providing, by the DNS-LB in response to a que- 
ry from the DNS-ISP, the IP address of a server 
based on the step of monitoring. 45 

A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
Internet service providers (ISPs) at a point of pres- 50 
ence (POP), each ISP having a load balancing do- 
main name server (DNS-ISP-LB), the servers fur- 
ther having an authoritative domain name server 
(DNS-A) associated therewith containing informa- 
tion regarding the IP addresses of the servers, the 55 
method comprising the steps of: 

deploying a first plurality of measurement serv- 



ice agents (MServices) in close physical prox- 
imity to the ISP POPs; 

monitoring, by the MServices at a location 
physically proximate to the ISP POP, perform- 
ance of the servers; and 

providing, by the DNS-ISP-LB in response to a 
query from the client, the IP address of a server 
based on the step of monitoring. 

22. A method of performing client-centric load balanc- 
ing of multiple globally-dispersed servers, the serv- 
ers being accessed by clients connecting through 
Internet service providers (ISPs) at a point of pres- 
ence (POP), each ISP having a load balancing do- 
main name server (DNS-ISP-LB), the servers fur- 
ther having an authoritative domain name server 
(DNS-A) associated therewith containing informa- 
tion regarding the IP addresses of the servers, the 
method comprising the steps of: 

deploying a first plurality of measurement serv- 
ice agents (MServices) in close physical prox- 
imity to the ISP POPs; 

monitoring, by the MServices at a location 
physically proximate to the ISP POP, perform- 
ance of the servers; and 

providing, by the DNS-ISP-LB in response to a 
query from the client, an IP address of the 
MService. 
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